Neural networks are computational systems that permit computers to essentially function in a manner analogous to that of the human brain. Neural networks do not utilize the traditional digital model of manipulating 0's and 1's. Instead, neural networks create connections between processing elements, which are equivalent to neurons of a human brain. Neural networks are thus based on various electronic circuits that are modeled on human nerve cells (i.e., neurons).
Generally, a neural network is an information-processing network, which is inspired by the manner in which a human brain performs a particular task or function of interest. Computational or artificial neural networks are thus inspired by biological neural systems. The elementary building blocks of biological neural systems are the neuron, the modifiable connections between the neurons, and the topology of the network.
Biologically inspired artificial neural networks have opened up new possibilities to apply computation to areas that were previously thought to be the exclusive domain of human intelligence. Neural networks learn and remember in ways that resemble human processes. Areas that show the greatest promise for neural networks, such as pattern classification tasks, speech and image recognition, are areas where conventional computers and data-processing systems have had the greatest difficulty.
In general, artificial neural networks are systems composed of many nonlinear computational elements operating in parallel and arranged in patterns reminiscent of biological neural nets. The computational elements, or nodes, are connected via variable weights that are typically adapted during use to improve performance. Thus, in solving a problem, neural net models can explore many competing hypothesis simultaneously using massively parallel nets composed of many computational elements connected by links with variable weights.
In contrast, with conventional von Neumann computers, an algorithm must first be developed manually, and a program of instructions written and executed sequentially. In some applications, this has proved extremely difficult. This makes conventional computers unsuitable for many real-time problems for which we have no efficient algorithm.
In a neural network, “neuron-like” nodes can output a signal based on the sum of their inputs, the output being the result of an activation function. In a neural network, there exists a plurality of connections, which are electrically coupled among a plurality of neurons. The connections serve as communication bridges among of a plurality of neurons coupled thereto. A network of such neuron-like nodes has the ability to process information in a variety of useful ways. By adjusting the connection values between neurons in a network, one can match certain inputs with desired outputs.
One does not program a neural network. Instead, one “teaches” a neural network by examples. Of course, there are many variations. For instance, some networks do not require examples and extract information directly from the input data. The two variations are thus called supervised and unsupervised learning. Neural networks are currently used in applications such as noise filtering, face and voice recognition and pattern recognition. Neural networks can thus be utilized as an advanced technique for processing information.
Neural networks that have been developed to date are largely software-based. A true neural network (e.g., the human brain) is massively parallel (and therefore very fast computationally) and very adaptable. For example, half of a human brain can suffer a lesion early in its development and not seriously affect its performance. Software simulations are slow because during the learning phase a standard computer must serially calculate connection strengths. When the networks get larger (and therefore more powerful and useful), the computational time becomes enormous.
For example, networks with 10,000 connections can easily overwhelm a computer. In comparison, the human brain has about 100 billion neurons, each of which can be connected to about 5,000 other neurons. On the other hand, if a network is trained to perform a specific task, perhaps taking many days or months to train, the final useful result can be built or “downloaded” onto a piece of hardware and also mass-produced. Because most problems requiring complex pattern recognition are highly specific, networks are task-specific. Thus, users usually provide their own, task-specific training data.
A number of software simulations of neural networks have been developed. Because software simulations are performed on conventional sequential computers, however, they do not take advantage of the inherent parallelism of neural network architectures. Consequently, they are relatively slow. One frequently used measurement of the speed of a neural network processor is the number of interconnections it can perform per second.
For example, the fastest software simulations available can perform up to approximately 18 million interconnects per second. Such speeds, however, currently require expensive super computers to achieve. Even so, approximately 18 million interconnects per second is still too slow to perform many classes of pattern classification tasks in real time. These include radar target classifications, sonar target classification, automatic speaker identification, automatic speech recognition, electrocardiogram analysis, etc.
The implementation of neural network systems has lagged somewhat behind their theoretical potential due to the difficulties in building neural network hardware. This is primarily because of the large numbers of neurons and weighted connections required. The emulation of even of the simplest biological nervous systems would require neurons and connections numbering in the millions and/or billions.
Due to the difficulties in constructing such highly interconnected processors, currently available neural network hardware systems have not approached this level of complexity. Another disadvantage of hardware systems is that they typically are often custom designed and configured to implement one particular neural network architecture and are not easily, if at all, reconfigurable in implementing different architectures. A true physical neural network chip, with the learning abilities and connectivity of a biological network, has not yet been designed and successfully implemented.
The problem with a pure hardware implementation of a neural network utilizing existing technology is the inability to physically form a great number of connections and neurons. On-chip learning can exist, but the size of the network is limited by digital processing methods and associated electronic circuitry. One of the difficulties in creating true physical neural networks lies in the highly complex manner in which a physical neural network must be designed and constructed. The present inventor believes that solutions to creating a true physical and artificial neural network lie in the use of nanotechnology and the implementation of a novel form of variable connections.
The term “Nanotechnology” generally refers to nanometer-scale manufacturing processes, materials and devices, as associated with, for example, nanometer-scale lithography and nanometer-scale information storage. Nanometer-scale components find utility in a wide variety of fields, particularly in the fabrication of microelectrical and microelectromechanical systems (commonly referred to as “MEMS”). Microelectrical nano-sized components include transistors, resistors, capacitors and other nano-integrated circuit components. MEMS devices include, for example, micro-sensors, micro-actuators, micro-instruments, micro-optics, and the like.
In general, nanotechnology presents a solution to the problems faced in the rapid pace of computer chip design in recent years. According to Moore's law, the number of switches that can be produced on a computer chip has doubled every 18 months. Chips now can hold millions of transistors. It is, becoming increasingly difficult, however, to increase the number of elements on a chip utilizing existing technologies. At the present rate, in the next few years the theoretical limit of silicon-based chips will likely have been attained. Because the number of elements and components that can be manufactured on a chip determines the data storage and processing capabilities of microchips, new technologies are required for the development of higher performance integrated circuit chips.
Present chip technology is also limited in cases where wires must be crossed on a chip. For the most part, the design of a computer chip is limited to two dimensions. Each time a circuit is forced to cross another circuit, another layer must be added to the chip. This increases the cost and decreases the speed of the resulting chip. A number of alternatives to standard silicon based complementary metal oxide semiconductor (“CMOS”) devices have been proposed. The common goal is to produce logic devices on a nanometer scale. Such dimensions are more commonly associated with molecules than integrated circuits.
The issue of interconnects in neural network hardware poses a serious problem. Because of the massive interconnectivity, a neural network constructed with standard integrated electronic methods can never reach the desired neuron and synapse density, simply because the interconnections overwhelm the largely 2-diminsional chip. It can thus be appreciated that almost any sort of 3-diminsional connectivity, no matter how simple, could offer tremendous benefits.
Integrated circuits and electrical components thereof, which can be produced at a molecular and nanometer scale, include devices such as carbon nanotubes and nanowires, which essentially are nanoscale conductors (“nanoconductors”). Nanoconductors are tiny conductive tubes (i.e., hollow) or wires (i.e., solid) with a very small size scale (e.g., 0.7 to 300 nanometers in diameter and up to 1 mm in length). Their structure and fabrication have been widely reported and are well known in the art. Carbon nanotubes, for example, exhibit a unique atomic arrangement, and possess useful physical properties such as one-dimensional electrical behavior, quantum conductance, and ballistic electron transport.
Carbon nanotubes are among the smallest dimensioned nanotube materials with a generally high aspect ratio and small diameter. High-quality single-walled carbon nanotubes can be grown as randomly oriented, needle-like or spaghetti-like tangled tubules. They can be grown by a number of fabrication methods, including chemical vapor deposition (CVD), laser ablation or electric arc growth.
Carbon nanotubes can be grown on a substrate by catalytic decomposition of hydrocarbon containing precursors such as ethylene, methane, or benzene. Nucleation layers, such as thin coatings of Ni, Co, or Fe are often intentionally added onto the substrate surface in order to nucleate a multiplicity of isolated nanotubes. Carbon nanotubes can also be nucleated and grown on a substrate without a metal nucleating layer by using a precursor including one or more of these metal atoms. Semiconductor nanowires can be grown on substrates by similar processes.
Attempts have been made to construct electronic devices utilizing nano-sized electrical devices and components. For example, a molecular wire crossbar memory device can be constructed from crossbar arrays of nanowires sandwiching molecules that act as on/off switches. Such a device can be formed from a plurality of nanometer-scale devices, each device comprising a junction formed by a pair of crossed wires where a single wire crosses another and at least one connector species connects the pair of crossed wires in the junction. The connector species comprises a bi-stable molecular switch. The junction forms either a resistor or a diode or an asymmetric non-linear resistor. The junction has a state that is capable of being altered by application of a first voltage and sensed by the application of a second, non-destructive voltage. A series of related patents attempts to cover everything from molecular logic to how to chemically assemble these devices.
Such a molecular crossbar device has two general applications. The notion of transistors built from nanotubes and relying on nanotube properties is being pursued. Second, two wires can be selectively brought to a certain voltage and the resulting electrostatic force attracts them. When they touch, the Van der Walls force keeps them in contact with each other and a “bit” is stored. The connections in this apparatus can therefore be utilized for a standard (i.e., binary and serial) computer. The inventors of such a device thus desire to coax a nanoconductor into a binary storage media or a transistor. As it turns out, such a device is easier to utilize as a storage device.
A need exists for a physical neural network, which can be implemented in the context of a semiconductor integrated circuit (i.e., a computer chip). Such a device, which can be referred to as a “physical neural network chip” or a “synapse chip” is thus disclosed herein. Such a device, if successfully implemented would be well suited for use with neural networks.
Researchers in the neuro-biological fields have attempted to develop a computationally efficient algorithm that can emulate a biologically realistic neural network. Specifically, researchers have attempted to develop method and/or systems, which would allow the efficient calculation of Spike-Timing Dependent-Plasticity (STDP), while also permitting fully interconnected pulsed networks. In STDP, timing between pre- and post-synaptic events can cause a net potentiation (LTP) or a net depression (LTD) of synapses.
A background for STPD is explained in “Mechanisms and Significance of Spike-Timing Dependent Plasticity” by Uma R. Karmarkar, et al., Biol. Cybern. 87, 373-382 (2002). STPD is also explained generally in “A Model of Spike-Timing Dependent Plasticity: One or Two Coincidence Detectors?” by Uma R. Karmarkar, et al., J. Neurophysiol, 88: 507-513, 2002. An additional reference regarding STDP is “Spike-Based Learning Rules and Stabilization of Persistent Neural Activity,” by Xiaohui Xie, et al., Dept. of Brain & Cog, Sci, MIT, Cambridge, Mass. A further reference regarding STDP and issues faced thereof is described in “Dendritic Spikes as a Mechanism for Cooperative Long-Term Potentiation,” Nace L. Golding, et al., Nature, Vol. 418, 18 Jul. 2002, pp. 326-330. The aforementioned references are indicated herein for generally illustrative, background and edification purposes only and do not disclose any features which limit any embodiments disclosed herein.
In the field of neural networks, Hebbian and anti-Hebbian learning has been the subject of great interest in recent years. An example of Hebbian learning is disclosed in non-limiting and non-essential U.S. Pat. No. 6,643,627, “Dynamic Synapse for Signal Processing in Neural Networks,” which issued to Liaw et al on Nov. 4, 2003, and which is incorporated herein by reference. Non-essential U.S. Pat. No. 6,643,627 is a non-essential reference with respect to embodiments of the present invention, and is referred to herein merely for general edification and background purposes only.
Another example of Hebbian learning is disclosed in non-limiting and non-essential U.S. Pat. No. 6,611,823, “Backlash Compensation Using Neural Network”, which issued to Selmic et al on Aug. 26, 2003. U.S. Pat. No. 6,111,823 also is a non-essential reference with respect to embodiments of the present invention, and is referred to herein merely for general edification and background purposes only. A further example of Hebbian learning and rules thereof is disclosed in non-essential U.S. Pat. No. 6,601,049, “Self-adjusting multi-layer neural network architectures and methods therefore”, which issued to David Cooper on Jul. 29, 2003, and is incorporated herein by reference. U.S. Pat. No. 6,601,049 is also a non-essential reference with respect to embodiments of the present invention, and is referred to herein merely for general edification and background purposes only.
To date, Hebbian and anti-Hebbian learning has been implemented utilizing software simulations and algorithms thereof. Some large scale physical neural network systems have also attempted to incorporate Hebbian and anti-Hebbian learning. Current Hebbian and anti-Hebbian learning systems have not been implemented via true physical neural networks, particularly networks developed at a molecular scale, such as physical neural networks implemented utilizing nanotechnology.